Al)  A  1 3r>  ?0R  STATISHCAL  OATA  PROCESSING  SYSIE#  MODI  l  [ NG  AND 
REl  IAB11  IlYtUl* STANFORD  UNIV  CA  !  KAILAIH  El  Al 
JUN  83  Af OSR ■ IR • 83 ■ 0935  F 49620  79  C  OOSfl 


i!N(.l  ASM  I  IF  D 


f  /g  n  n 


Thoms  Kailath,  Robert  N.  Gray,  Abbas  El  Ganal 
Martin  Morf 


scnroMMtHa  oaoanization  hams  and  a 


F49620— 7 9-C-0058 


Stanford  University 
Stanford,  California  94305 


tt.  CONTMOLLINO  OFFICC  NANS  AND  ADDNUS 


AFOSR/l/Af 
Bolling  AFB 


l .  ill  lllil  •  ivTT^M  '14Mf -1'  Wr 


I  Sunni  Irani  CMlralftaS  Ottlam)  IS.  MCUAlTV  CLASS,  (ml  thlm  nmort) 

UNCLASSIFIED 


i  rm  \  rrriTEirm  rv  2  rr 


Approved  for  public  release 
distribution  unlimited 


UNCLASSIFIED 


.* 


,/wtCUMTV  CIAWHCATWH  OW  THt Ml  MmtZi* 


S.  Reliable  VLSI  Computing  Structures 

4.  Algorithms  and  Architectures  for  Statistical  and  Data  Processing 


Aeeeaslen  ?or 

IHIS  GRAM 
DIIC  TAB 
Unannounced 
Justification. 


ff 


AFOSR-TR-  83-0  935 


May  1,  im  -  Jim  SO,  IMS 


AbbaaElGamal 
Martfai  Morf 


The  aim  of  research  described  herein  was  explore  several  fundamental  prob¬ 
lems  in  statistical  data  processing  and  system  modeling,  with  particular  aspects! 


~Tfite~results  obtained  hare  almost  completely  bear  described  in  published  journal 
and  conference  papers,  so  this  report  will  focus  on  a  summary  of  major  results. 

The  presentation  will  be  in  separate  sections  corresponding  to  the  efforts  of 
the  co-principal  investigators  -  T.  Kailath,  R.  Gray,  A.  El  Gama!  ami  M.  Morf: 

Analysis  of  Nonstatioaary  Signal  Processes  j 
Algorithms  for  Data  Compression  J 
9-Ar-  Reliable  VLSI  Computing  Structures  j1 

Algorithms  sad  AjrehHectnrm  for  Statistical  and  Data  ProcessiBg,^^ 


Ths  goal  of  this  project  has  beea  to  develop  a  variety  of  signal  processing 
aUsrithmn  with  emphasis  oa  aoastaUsnary  procseesa,  reduced  eomputatiouary 
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1.  Characterization  and  iumditiiffai  of  ten stationary  processes  based  on 
the  concept  of  displacement  node. 

2.  Developing  new  algorithms  for  a  variety  of  problems  in  array  processing, 
spectral  estimation,  and  adaptive  filtering. 


The  notion  of  displacement  rank  of  a  covariance  was  utilized  to  construct 
efficient  par ametrizat ions  of  nonstationary  discrete-time  second-order  processes. 
Several  distinct  parametrizations  were  derived,  ill  sharing  a  common  set  of  Sehwr 
coefficient^  which  are  a  generalization  Of  the  wefl  known  reflection  (or  partial 
correlation)  coefficients  associated  with  stationary  processes.  The  Schur 
coefficients,  and  an  additional  set  of  tapped  drisy  line  eoeikfaots,  serve  as  gains 

4 

in  lattice-form  modeling  end  whitening  liters  for  ponst  alien  nry  pspeenn.  These 
lattice  filters  consist  of  coactant  paramHtr  sections;  the  oafer  time-variation 
required  is  the  growth  in  time  of  the  liter  order. 

The  of  wHdteuty  pTccmess  fifth  finite  dtspkeesasot  rank 

illwninated  the  tignilrencs  and  vsesatOty  of  the  Sekmr  rsdesWst. procedure. 
This  analytical  procedure,  originally  fermulufl  by  Schur  (N77)  for  funetiou- 
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Chief,  Tee  hoi  eel  Information  Division 
as  to  clarify  the  rebtion  between  the 


noestat  tonary  eovarianefc  Oof  analyst  server  to  clarify  the  nihtfc*  between  the 
numerous  nMotioo  of  the  procedure  and  to  «t»4  it  to  mnttkhannei  cases. 


It  also  spurred  a 


resnits  in  ths  theory  erf  inverse  scattering. 


Several  specific  efeases  of  nowtatkmsry  processes  emerged  as  generafisatioas 
of  stationary  precwwe:  Oeeafofatfo wary  protwu,  which  are  obtained  by  linear 
time  invariant  £,11)  filtering  «*  stationary  proccaem,  pomes  the  same  structural 
properties  (including  the  same  lattice  models)  as  stationery  feroeemw.  Disripative 
processes,  which  include  quasi-stationery  as  a  particular  case,  have  lattice  models 
of  higher  compkxity,  but  at®  abate  many  properties  of  stationary  covariances. 


Another  outcome  of  the  analyric  of  nonstetioasry  ptocesses  is  the  formula- 
tioa  of  a  unified  theory  of  spectral  analysis  for  sueh  processes.  Oar  results 
include  as  particular  cases  thepretfou#  developed  theory  of  asymptotically  star 
rionnry, '  asymptotically  mean  stationary  and  harmbnisable  processes.  O® 
analysis  provides  also  sn  attract*?#  alternative  to  Wiener’s  generalised  harmonic 
amaijak  for  a  broad  dam  of  nonetmkmary  pretussm. 

Several  papers  lave  been  written  on  time  topics,  eo-au&orod  with  R  f*orat 

and  H.  Lev*Atf,wlth  *>  ^E1EE  m 

kfitmm*  neevy  ii  Jhnneryffifirfsndln  Ssptri^sr  IttM  and  another  paper  in 
m  ZKS#  TVrtiuiiffinn  eit^efiniKfib  Cbnhmr  m  Oetober  IMS.  ’ 
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adaptiyohltariag  sppheettoae,  such  asrhsahd  eqaafaattea  Hi  echo  eaaeeBsr 
tioa,  the  order  of  the  desired  sstteatfea  fehaows  or  ap  psr^oosded  a  priori  It 
these  esses,  the  lower  order  w#ni>  are  Ml  resHjr  aoesssarj,  sod  farther 


these  esses,  the  fewer  order  are  oak  rsafljr  oeesssarj,  sod  farther 

sigaifleaat  computatioaal  sad  wenplileetfews  result  whea  the 

adaptive  liter  directly  ooapotse  thato«±<wde»  rwidusle  reeereively  ia  twee.  We 


hare  developed 


jaew  feat,  toed  order. 


tapped-delsy-liae  adaptive  4Uteriag  that  algorithasa  retake  fewer  operatfeas  per 


iteraifea  sad  exhibit  better 


that  the 


Kahnaa  algorithm  of  Morf,  Ljeagaad  Fafeeaer  {MU).  Ia  wparira  with  the 
stochaetfe-gradieet  or  LMS  adaptive  algorithm  ofWidrow  aad  HoK,  the  sew  aew, 
fixed-order,  least-squares  algorithms  yield  sabetsatlal  toproveweafe  is  traasfeat 


behavior  at  a  moded 


is  eomputatioaal  complexity.  AddttfeaaBy,  over  a 


simple  modification  of  the  new  algorithms,  and  aifiitriiy  weighting  of  the 
influence  of  these  initial  conditions  through  a  soft-constraint  ia  also  permitted  to 
tednee  the  effects  of  noise  npon  a  good  initial  condition. 

In  related  work,  an  efficient  exact-least-sqnares  procedure  was  derdoped  for 
the  adaptive  adjustment  of  «  fractionally  spaced  equaliser  (FSE).  Interaymbol 
interpolation  of  As  desired  training  sequence  is  need  by  this  new  procednre  to 
reduce  computational  requirements  and  to  improve  convergence.  For  a  T/p  FSE, 
a  factor  of  p  improvement  in  “start-op”  tune  is  etteined  alive  to  the  multichan¬ 
nel  FSE  versions  of  the  least-equares  algorithms  of  Falconer  aid  Ljung  [1978J  and 
of  Satorius  and  Pack  (1981}.  Additional  reductions  in  oompatetjonsl  require¬ 
ments  ere  achieved  by  a  special  ex  act-leas  t-equares  modHkation  for  the  paasband 
“Nyquist”  FSE  structure  of  MueBer  and  Werner.  Tie  procednre  Is  shown  to  he 
most  efficiently  implemented  using  a  tfonsvssinl  Iff  or  reafization  of  the  test 

the  new  procedure  (T/4  FSE)  is  found  to  ho  approntmaUiy  t he  Mtffei  as  that  of 
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2.  Algorithms  for  Dots  Compression  (R.  M.  Gray) 

Research  Objectives 

Hie  general  goal  of  this  project  has  been  to  develop  computer-aided  design 
algorithms  for  data  compression  sptepv  aad  to  study  the  relative  performance, 
complexity,  sad  rate  of  such  systems.  Where  possible,  comparisons  have  also 
been  made  with  theoretical  bounds  and  with  traditional  approaches.  Particular 
emphasis  has  been  placed  on  vector  quantisation  (vector  coding,  block  coding, 
multidimensional  quantization)  systems  lor  speech  waveforms,  for  linear  predie* 
tive  speech  parameters,  and  for  various  random  processes. 

Major  Accomplishments 

As  detailed  in  the  annual  reports  and  in  the  following  fist  of  publications, 
the  research  effort  has  been  exceedingly  fruitful  in  laying  the  groundwork  for 
computer  aided  design  of  a  variety  of  data  compression  systems.  The  algorithm 
of  Linde,  Buzo,  and  Gray  (1980)  for  the  design  of  locally  optimum  vector  quan¬ 
tizers  which  was  developed  with  the  support  of  this  contract  has  been  extended 
both  by  our  Stanford  group  had  by  n  several  other  institutions  to  develop 
moderate  complexity  low  rate  and  very  four  rate  data  rnsapmssina  systems  for 
speech  waveforms,  voice  coders,  various  random  proeesms,  and,  most  recently, 
images.  Our  group  pioneered  the  baric  algorithm  aad  two  of  Us  most  important 
variations:  tree  searched  codes  and  product  codes.  Is  adilHon  to  the  results  of 
this  research  project,  rim  haste  tiirhniquss  Jrvrispii  hy  this  project  have  been 
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speech  recognition  systems  baaed  on  vector  quantization  and  not  requiring 
dynamic  time  warping  have  been  developed  at  the  Naval  Research  Laboratory, 
the  University  of  Mexico,  Osaka  University,  and  Bell  Laboratories  using  varia¬ 
tions  on  our  algorithms.  New  low  complexity  image  coding  systems  of  rates  less 
than  one  bit  per  pixel  have  been  developed  using  our  algorithms  at  the  University 
of  California,  by  our  group,  and  by  Mitsubishi  Corp. 

Part  of  the  accomplishment  of  this  project  was  the  development  of  extensive 
software  for  vector  quantizer  design  and  data  compression  simulations.  This 
software  has  been  shared  and  extensively  used  by  the  Naval  Research  Labora¬ 
tory. 

During  the  final  year  of  this  contract  the  emphasis  has  been  on  the  develop¬ 
ment  of  shape/gain  vector  quantizers,  an  example  of  product  code  quantization 
systems  that  operate  in  a  memoryless  fashion  on  successive  data  vectors  and 
separately  quantize  a  scalar  gain  i  n  and  a  vector  shape  term.  The  quantiza¬ 
tions  are  coupled  by  the  distortion  measure  so  that  the  encoder  is  optimal  for 
codes  with  this  structure.  An  iterative  improvement  algorithm  was  developed  to 
yield  locally  optimum  codes  of  the  desired  structure.  The  goal  of  this  style  of 
code  is  twofold:  To  provide  better  dynamic  range  by  separately  treating  the 
energy  and  to  provide  a  means  of  designing  higher  rate  and  hence  better  quality 
vector  quantizers  with  reasonable  computational  complexity  and  memory  require¬ 
ments.  These  codes  are  capable  of  providing  better  performance  for  a  fixed  rate 
and  complexity  than  the  ordinary  high  complexity  vector  quantizers.  Prelim¬ 
inary  results  tor  these  systems  were  presented  by  Saba  and  Gray  (1983)  and  a 
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paper  bee  recently  bee*  submitted  for  submission  for  publication.  A  copy  of  this 
paper  will  be  forwarded  to  the  Air  Force  when  complete  as  an  epilogue  to  this 
final  report 

With  the  exception  of  tile  final  paper  bang  prepared,  all  of  the  principal 
research  results  developed  during  the  course  of  this  project  have  been  reported  in 
the  open  literature.  Preliminary  portions  of  the  find  paper  may  be  found  in  the 
conference  proceedings  of  the  paper  by  Salmi  and  Gray  (1983). 

With  tiie  termination  of  this  contract,  the  image  coding  and  the  speech  cod¬ 
ing  and  recognition  work  will  continued  with  the  support  of  the  Army  Research 
Office  and  the  Joint  Services  Electronics  Program.  Unlike  the  research  reported 
here,  the  future  research  will  focus  on  feedback  vector  quantisers  and  finite-state 
vector  quantisers  instead  of  the  memoryieas  victor  quantiser  of  this  project.  LSI 
and  VLSI  implementations  of  vector  quantitation  systems  with  and  without 
memory  will  be  continued  with  the  support  of  the  Joint  Services  Electronics  Pan- 
gram. 
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A.  Buzo,  A.  H.  Gray,  Jr.,  R.  M.  Gray,  and  J.  D.  Market,  “Speech  coding  based 
upon  rector  quantisation,”  IEEE  Tranaaetione  on  Aconetiea,  Speech,  and  Signal 
Processing,  VoL  ASSP-S8,  pp.  588*574,  October  1080. 

R.  M.  Gray,  A.  H.  Gray  Jr.,  G.  Rebottedo,  and  J.  B.  Shore,  “Rate  distortion 
speech  coding  with  a  minimum  discrimination  information  <ttstortion  measure,” 
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A.  Be  Our,  kagRaUe  Graft  *4  I.  D.  MuM, 


I.OO. 


’■  ... 


R.  M.Gray,  “TmmwM  block  mm  com,”  mmiift  1990  Antrim 
Conference,  ADertoa,  IL,  Oct.  1960. 

HL  Abet,  R.  M.  Gray,  nd  G.  RAolhdo,  “Vector  gputtaUf  of  speech  and 
qwe^fike  emlNM,*  iMf  OSES  fn fsmeffoaaf  Spmpeeiom  on  fnformetion 
Theory,  State  Monica,  Feb.  1961. 

G.  Rebottedo  tad  R.  hi.  Gray,  “Vector  ftattelka  apptted  to  speech  coding," 
Ibid. 


H.  Abat,  R.  M.  Gray,  tad  G.  Rebottedo,  “Vector  quantisation  of  speech 
waveforms,”  1981  hdemoUonel  Conference  ea  Aeematice,  Speech,  end  Siptal  Pn- 
eesstuy,  Atalaata,  GA,  April  1961. 

G.  Rebolledo,  R.  M.  Qrqr,  tad  J.  P.  Barf,  “A  residual  netted  Xaear  predictive 
vocoder  based  apet  vector  qasatiistiaa,”  Msdetadh  Guadalajara,  Mexico. 


M.  J.  Sabin  and  R.  M.  Gray,  “An  algorithm  for  tee  design  of  product  code  vector 
quantizers,”  1989  IEEE  hiemetienel  Spnpoehm  on  Informetion  Theory,  Lee 
Arcs,  France,  Jam  ML 


M.  J.Sabia  and  R.  M.  Gray,  “Product  code  vector  qa—titssp  for  speech 
waveform  coding,"  Conference  Record,  GUheeom  ' 89,  pp.  1067-1091,  December 
1968. 


Papers  Under  Review 


M.  J.  Sabin  and  R.  M.  Gray,  “Product  code  vector  quantizers  for  waveform  and 
voice  coding,”  IEEE  Treat,  mASSP,  lane  1968. 


€  r  '«r'.  V  i 


»*S>  ,! veiu*S«i'»\*  fi 6  ■»  ' 


V  •  jA 


,  tl  t-H.it)  W  ».A  •  / 


-IT- 


S.  Reliable  VLSI 


Research  Objectives 


The  following  topics  have  bees  under  investigation:  (i)  Coding  for  memories 
with  stuck-at  defects  sad  rsadom  errors  whea  the  defect  information  is  provided 
to  the  encoder  or  to  the  decoder,  (ii)  The  improvement  ia  storage  capacity 
achieved  by  skipping  defective  or  tome  partially  defective  memory  cells.  (iii)  The 
complexity  of  encoding  aad  decoding  circuitry,  (iv)  The  area  and  delay  penalties 
involved  in  structuring  VLSI  arrays,  (v)  Communication  complexity  of  comput- 


(i)  Heegard  [1]  has  examined  a  class  of  linear  block  codes  (LBC)  for  improv¬ 
ing  the  reliable  storage  of  information  ia  a  computer  memory  with  stuck-at 
defects  and  noise.  Re  examined  LBCY  when  the  “side*  information  about  the 
state  of  the  defects  ia  available  to  toe  decoder  or  to  the  encoder.  In  the  former 
case,  stuck-at  ceOs  act  as  erasures  se  that  taekaiqaes'for  decoding  the  LBCY  with 

ii  *  ;  f 

erasures  and  error*  end  be  engjfojrod.  R*  introduced  a  rites  of  modifted  linear 
block  codes  (M.BCYJ  to  correct  dthcfci  and  errors  when  the  location  and  nature 
of  the  defects  Is  given  to  the  encoder.  # 

S  '  _  h 

ttnrm  1  of  |1]  rhsrsrteriim  tbs  defect  and  arror  correction  tapabittty  of 
LBCY  wto  Id^Cib  tontflf  nitoHndhtaaim  .i ;  .  * 
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A  elan  of  modified  ayefie  codes 


in  (!}.  The  BCH  boned  for 


these  cyclic  codes  was  derived  and  employed  to  construct  MLBC’s  with  specified 
bounds  on  the  mm  imam  distances. 

(ii)  la  {!},  the  proposer  aad  Greene  considered  a  memory  composed  of  AT 
discrete  crib,  each  characterised  by  a  defect  state  <  drawn  independently 
according  to  p(t) .  The  probability  of  retrieving  a  symbol  y  given  •  and  the 
stored  symbol  s  is  completely  specified  by  p(y  |s,«) .  The  selector  identifies  a 
subset  of  "good*  cells,  which  alone  are  nsed  to  store  data,  in  an  effort  to  improve 
the  reliable  storage  rate  of  the  memory. 


ENCODER/  SELECTOR  MEMORY 
DECODER  . _ . 


DATA  ♦, 
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•>  19  b&i  ■  "  . 
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mam  if  wrier  =>  '">■»*  hviar^Ki 


r- 
for; 


%  » 1  if  and  only  if  the  i*  cell  fa  used.  The  symbols  are  stored  in 

order  in  the  selected  cells.  A  storage  rate  R  is  achieved  if  there  exfats  a 
sequence  of  (2*s,rN)  codes,  selection  rales  p(u  |s)  and  decoding  rales  such  that 
the  probability  error  tends  to  zero. 

The  storage  capacity  fa  established  for  independent  selection  rules 

N 

p(u  |«)  ■  J]  p(«,- 1 *i)  .  It  fa  then  shown  that  the  capacity  fa  higher  for  the  more 

»— i 

rr 

general  class  of  causal  rules  p(u  |s)—  p(«,|«, . t{)  .  However,  for  the  cell 

i— t 

consisting  of  two  binary  symmetric  channels  (BSC’s),  the  capacity  for  causal  rules 
fa  achieved  by  an  independent  rale.  A  similar  result  hold  for  any  two-state  cell 
when  the  state  fa  known  to  the  decoder. 

For  arbitrary  selection  rules,  rates  higher  than  times  possible  with  causal 
rules  are  achievable,  even  for  two-state  BSC  cells.  The  capacity  for  arbitrary 
rules  fa  as  yet  unknown. 

(in)  Ahlswede  has  proved  the  Elias  [3]  -  Winograd  [4]  result  for  probability 

of  error  criterion.  The  capacity  -»  •  as  — r-r  ,  where  k  fa  the  number  of 

c  log  s 

information  bits,  (results  not  written  up). 

(far)  b  (6),  the  proposer  and  Greene  investigated  the  asymptotic  penalties  of 
restructuring  homogeneous  VLSI  arrays  for  yield  enhancement  Each  dement  of 
the  fabricated  array  fa  ««h9pd  to  be  defective  with  independent  probability  p  . 
A  feed  fraction  R  of  the  dements  ate  to  be  connected  into  a  preepeciled  regu- 


of  the  longest  connection  sad  #  be  the  number  of  wiring  tracks  needed  to 
accomplish  the  interconnection.  It  is  shown  that: 

(1)  Connecting  a  chain  of  K  elements  from  a  linear  array  of  N  elements 
requires  4  —  Q(logN)  and  t  -■  1  track  running  parallel  to  the  array. 

(2)  Connecting  a  linear  array  of  K  fixed  I/O  ports  to  distinct  non¬ 
defective  elements  from  a  parallel  array  requires  4  “  0(/*#N)  and 
%  -  (T(logN) . 

(3)  Connecting  K  elements  from  aa  N-element  linear  array  to  K  from  a 
parallel  N-element  array,  in  pairs,  requires  only  constant  4  and  t  . 
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(4)  Coaaecting  a  ekta  of  K  WKrti  aom  »  SXX  array  requires  con- 
staat  4  and  om  trade  titam  stones tr,  tUi  problem  is  closely 
related  to  the  percolation  problem  of  statktieal  physics. 

la  dt  the  abort  eases,  alphas  adhering  the  bauds  on  4  iad  t 
ate  presented  which  canaeel  tbs  array  with  probabilAy  approaching 

ode.  The  algorithms  ran  in  d<«)  time. 

}  r 

(5)  Connecting  a  KXK  tqaare  lattice  from  an  WXN  array  k  shown  to 
require  d  —  OVtCpfJ  .  la  (0),  It  is  shows  that  with  4  —  OV{UfN) 
onfy  a  constant  number  of  tradu  are  seeded.  The  proof  employs  the 
max  imam  flow-minimum  cat  theorem  for  graphs  with  random  capaci¬ 
ties. 

(v)  In  (7),  the  proposer  proved  that  if  (X,Y)  are  two  finite  alphabet  corre¬ 


lated  sources  with  y(s,y)  >  S  for  afi  (X  X  Jf) ,  and  if  a  function  F(X,Y) 

is  o-sensithre,  then  the  rattflt  of  teimibhiioa  firom  X  to  Y  necessary  to  compute 
F(X,Y)  reliably  mast  be  gnder  tha^er  eqaal  to  J#(X|  r) .  The 
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elements  f  €  Jf  with  >4  iadiftreat  from  one. 
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4.  Algorithms  and  Architectures  for  Statistical  and  Data  Processing 
(M.  Mori) 

Professor  Morf  has  moved  to  Yale  University.  He  will  be  sending  his  portion 
of  the  final  report  directly. 

However,  a  list  of  Ph.D.  students  partially  supported  under  this  contract, 


and  a  list  of  publications  in  the  contract  period  are  given  here. 
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tions  for  Continuous-Time  Linear  Estimation  Problems,”  IEEE  Transactions  on 
Automatic  Control  ( Accepted,  rauissd  seed  resubmitted) 
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